It was actually a term of formality very much like "Mister", "Mrs.", "Miss" or "Ms."
Especially since "brother" or "sister" was followed by the person's last name. It was awkward to use in a congregation with a dozen "Sister Jones". Neither endearing nor practical. It sounds very institutional. Calling on someone from the platform you'd get, "Sister Jones, Mary." I assume they didn't use "Sister Margret" or "Brother David", because it sounds too Catholicy.
In my family, if my sister was named "Elizabeth Jones," I wouldn't call her either "Sister Jones" or "Sister Elizabeth". Of course, I'd call her "Lizard Breath". For aunts and uncles, it would be their first names prefaced with "Aunt" or "Uncle". For grandparents, it was a grab bag.